feat(llmobs): support manual agent versioning - #9808
Conversation
Overall package sizeSelf size: 8.14 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f95627e590
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // only agent spans carry the tag. The span processor materializes it. | ||
| // Annotation context wins over the span's own option, matching `name` above. | ||
| const agentVersion = annotationContext?.agent?.version ?? version | ||
| if (agentVersion) this.tagAgentVersion(span, agentVersion) |
There was a problem hiding this comment.
When version is supplied on a non-agent trace/wrap call, #extractOptions warns that it is being ignored, but this path still passes the value into tagAgentVersion; a JavaScript caller using a numeric version such as llmobs.trace({ kind: 'workflow', version: 3 }, ...) will throw during span activation even though the span can never emit agent_version. Gate this validation to agent spans (or drop non-agent versions before reaching the tagger) so unsupported span kinds do not abort the application path.
AGENTS.md reference: AGENTS.md:L222-L225
Useful? React with 👍 / 👎.
| // only agent spans carry the tag. The span processor materializes it. | ||
| // Annotation context wins over the span's own option, matching `name` above. | ||
| const agentVersion = annotationContext?.agent?.version ?? version | ||
| if (agentVersion) this.tagAgentVersion(span, agentVersion) |
There was a problem hiding this comment.
When version is supplied on a non-agent trace/wrap call, #extractOptions warns that it is being ignored, but this path still passes the value into tagAgentVersion; a JavaScript caller using a numeric version such as llmobs.trace({ kind: 'workflow', version: 3 }, ...) will throw during span activation even though the span can never emit agent_version. Gate this validation to agent spans (or drop non-agent versions before reaching the tagger) so unsupported span kinds do not abort the application path.
Useful? React with 👍 / 👎.
BenchmarksBenchmark execution time: 2026-08-13 15:39:42 Comparing candidate commit 8ea014f in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2307 metrics, 51 unstable metrics.
|
🎉 All green!🧪 All tests passed 🔄 Datadog retried 1 test - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 8ea014f | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9808 +/- ##
=========================================
Coverage 98.51% 98.51%
=========================================
Files 975 975
Lines 142359 142405 +46
Branches 13882 12369 -1513
=========================================
+ Hits 140247 140293 +46
Misses 2112 2112
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8ea014feb4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| this._tagger.tagToolDefinitions(span, toolDefinitions) | ||
| } | ||
| if (agent?.version != null) { | ||
| this._tagger.tagAgentVersion(span, agent.version) |
There was a problem hiding this comment.
When llmobs.annotate() is called on a non-agent span with agent.version, this call validates the version even though the span processor later drops _ml_obs.agent_version for every non-agent kind. A JavaScript caller such as llmobs.trace({ kind: 'workflow' }, () => llmobs.annotate({ agent: { version: 3 } })) therefore throws in the application path for a value that cannot be emitted; the already-commented trace/wrap path has the same root, and this annotate surface is a separate public entry point that needs the same kind gate or drop-before-validation behavior.
AGENTS.md reference: AGENTS.md:L222-L225
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Adds manual agent versioning to the LLM Observability SDK, porting the Python support from DataDog/dd-trace-py#19490.
A user-supplied version can now be attached to an agent span through any of the four annotation surfaces:
Each emits a single
agent_versionspan tag, on agent spans only — never on their children. This matches the Python tag name and semantics exactly, so the backend rollups work identically across SDKs.Implementation notes:
_ml_obs.agent_versionkey at registration/annotation time, and only materialized into theagent_versiontag inLLMObsSpanProcessor#getTags, once the final span kind is known. This is required rather than tagging eagerly:annotationContextspans a whole block of spans of mixed kinds, and integrations may still callchangeKindat finish (e.g.plugins/claude-agent-sdkpromotes tool spans toagent), so the kind isn't settled until finish.versionoption both supply a value, the annotation context wins — consistent with the existing precedence fornameinregisterLLMObsSpan.versionis destructured out in#extractOptionsso it does not leak into the APM span options, and a warning is logged if it's supplied on a non-agent span (mirroring the Python decorator behavior).index.d.tsandindex.d.v5.tsper the dual-surface rule:Agentinterface,agent?: AgentonAnnotationOptionsandAnnotationContextOptions,version?: stringonLLMObsSpanOptions.Motivation
Part of the Agent Tracking work (M6). Users need to correlate agent behavior, cost, and evaluation results with the specific version of the agent that produced them. dd-trace-py shipped this in #19490; this brings Node.js to parity. A companion Go PR is open against dd-trace-go.
Additional Notes
Tested with four end-to-end tests in
packages/dd-trace/test/llmobs/sdk/integration.spec.js, asserting on the emitted payload: the span-option path, theannotatepath, the annotation-context path (agent span tagged, child tool span untagged), and the non-agent path (dropped from both entry points). Full regression sweep across the related llmobs specs (span_processor,tagger,sdk/index,sdk/integration,sdk/agent_attribution) is green at 338 passing.Claude session:
f18b59fb-588d-4ea7-a2bd-752e23d7b426Resume:
claude --resume f18b59fb-588d-4ea7-a2bd-752e23d7b426